Skip to content

feat: demo mode for showing a trained policy off live#75

Open
ZouzouWP wants to merge 1 commit into
huggingface:mainfrom
ZouzouWP:feat/demo-mode
Open

feat: demo mode for showing a trained policy off live#75
ZouzouWP wants to merge 1 commit into
huggingface:mainfrom
ZouzouWP:feat/demo-mode

Conversation

@ZouzouWP

Copy link
Copy Markdown

Showing a trained policy to someone (recruiters, teammates, a booth) meant re-entering the full inference config (checkpoint, robot, cameras) from scratch every time, with no simple live camera preview separate from the full recording UI.

What changed

  • New /demo page: one click loads the last successfully started inference config (policy, robot, cameras) as a preset from localStorage — no re-configuration needed right before a demo.
  • New lelab/rollout_frames.py: a thin wrapper around lerobot.scripts.lerobot_rollout that runs inference exactly as before, but also tees each camera frame to a JPEG on disk (~10fps). Served through a new /demo-camera/{cam_id} endpoint for a live, low-latency preview on the demo page.
flowchart LR
    A["InferenceModal:\nstart inference"] -->|saves preset| B["localStorage\npolicy + robot + cameras"]
    B --> C["/demo page\none-click preset"]
    C --> D["lerobot_rollout subprocess\nvia rollout_frames.py wrapper"]
    D -->|tees JPEG ~10fps| E["%TEMP%/lelab_demo_frames"]
    E --> F["GET /demo-camera/cam_id"]
    F --> G["Live preview\non /demo page"]
Loading

Dead end tried and reverted

Real-Time Chunking (--inference.type=rtc) was tried to remove a periodic ~200ms stall from ACT's chunk recompute. This lerobot version's RTC engine calls predict_action_chunk(inference_delay=...), which ACTPolicy doesn't accept — RTC targets flow-matching policies (pi0, smolvla), not ACT. Reverted cleanly to the standard sync engine; left a comment in the code for whoever revisits ACT-specific prefetching.

Testing

Run against a checkpoint trained on ~250 episodes: verified the live preview updates (279 frame updates observed in one run) and that the preset correctly restores policy/robot/cameras without manual re-entry.

Showing a trained policy to someone (recruiters, teammates, a booth) meant re-entering the full inference config (checkpoint, robot, cameras) from scratch every time, with no simple live camera preview separate from the full recording UI.

## What changed

- New `/demo` page: one click loads the **last successfully started inference config** (policy, robot, cameras) as a preset from `localStorage` — no re-configuration needed right before a demo.
- New `lelab/rollout_frames.py`: a thin wrapper around `lerobot.scripts.lerobot_rollout` that runs inference exactly as before, but also tees each camera frame to a JPEG on disk (~10fps). Served through a new `/demo-camera/{cam_id}` endpoint for a live, low-latency preview on the demo page.

```mermaid
flowchart LR
    A["InferenceModal:\nstart inference"] -->|saves preset| B["localStorage\npolicy + robot + cameras"]
    B --> C["/demo page\none-click preset"]
    C --> D["lerobot_rollout subprocess\nvia rollout_frames.py wrapper"]
    D -->|tees JPEG ~10fps| E["%TEMP%/lelab_demo_frames"]
    E --> F["GET /demo-camera/cam_id"]
    F --> G["Live preview\non /demo page"]
```

## Dead end tried and reverted

Real-Time Chunking (`--inference.type=rtc`) was tried to remove a periodic ~200ms stall from ACT's chunk recompute. This lerobot version's RTC engine calls `predict_action_chunk(inference_delay=...)`, which `ACTPolicy` doesn't accept — RTC targets flow-matching policies (pi0, smolvla), not ACT. Reverted cleanly to the standard sync engine; left a comment in the code for whoever revisits ACT-specific prefetching.

## Testing

Run against a checkpoint trained on ~250 episodes: verified the live preview updates (279 frame updates observed in one run) and that the preset correctly restores policy/robot/cameras without manual re-entry.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant